1. Field of the Invention
The present invention relates to a technique of suppressing deterioration of image quality caused by a foreign substance adhering to the surface of an optical low-pass filter or the like in an image capturing apparatus using an image sensor such as a CCD sensor or CMOS sensor.
2. Description of the Related Art
When the lens is detached from the camera body of a lens-interchangeable digital camera, dust floating in the air may enter the camera body. The camera incorporates various mechanical units such as a shutter mechanism which operate mechanically. The operation of these mechanical units may generate dust such as metal powder in the camera body.
When a foreign substance such as dust of any type adheres to the surface of an optical low-pass filter which is an optical element arranged in front of an image sensor and forms the image capturing unit of a digital camera, the shadow of the foreign substance is captured within the image, deteriorating the image quality.
A camera using silver-halide film feeds the film for each shot. Hence, images never contain the shadow of the same foreign substance at the same position continuously. However, a digital camera requires no operation to feed the film frame for each shot, and therefore, captured images may continuously contain the shadow of the same foreign substance at the same position.
To solve this problem, there is proposed a method of correcting a pixel that captures the shadow of a foreign substance by using the signals of neighboring pixels or the like. As a technique of correcting such a pixel, for example, Japanese Patent Laid-Open No. 6-105241 proposes a pixel defect correction method of correcting the pixel defect of an image sensor. Japanese Patent Laid-Open No. 2004-242158 proposes a method which simplifies setting of position information of a pixel defect. According to this method, the extension of an image file recorded in a dust acquisition mode is changed from that of a normal image. The PC (Personal Computer) automatically discriminates a dust information image, and corrects a target image based on this information.
Recently, there is proposed a technique of handling moving image information as digital data and encoding it at high compression rate with high image quality to accumulate and transmit it. This technique is becoming popular.
Motion JPEG (Joint Photographic Experts Group) encodes a moving image by applying still image encoding (e.g., JPEG encoding) to each frame. Although JPEG encoding basically targets still images, products which apply JPEG encoding to even moving images by high-speed processing have come into practical use.
H.264 (MPEG4-Part10 AVC) is an encoding method aiming at higher compression rates and higher image quality. It is known that H.264 requires more intensive computation for encoding and decoding than those in conventional encoding methods such as MPEG2 and MPEG4, but can achieve higher coding efficiencies (see ISO/IEC 14496-10, “Advanced Video Coding”).
Compact digital cameras capable of recording a moving image based on these encoding methods have also been developed and commercialized. Users can easily view images with such a digital camera, a personal computer, a DVD player, and the like.
In this situation, a need is recently growing for recording higher-resolution moving images with a larger number of pixels by lens-interchangeable digital cameras as well as compact digital cameras. However, as described above, due to a variety of factors lens-interchangeable digital cameras suffer the problem of dust adhering to the surface of an image sensor. If the lens-interchangeable digital camera records a moving image while dust adheres to the surface, the shadow of dust may always appear at the same position during moving image playback.
According to a conventional dust removal method based on image processing for the lens-interchangeable digital camera, information (e.g., information on the position and size of dust) necessary for dust removal and image data are recorded. The image is loaded later into a personal computer or the like to remove the shadow of dust by interpolation processing using pixels around the dust region. That is, the recorded image data contains the shadow of dust. As for still images, dust removal is executed for each one. As for a moving image, dust removal must be performed for the full recording time.
A dust removal method for a still image interpolates a dust shadow-containing region using neighboring pixels. At this time, even if an image around the pixels of the dust region is not uniform, a resultant image becomes unnatural at the boundary between the interpolated dust region and its surrounding region. In contrast, when removing the shadow of dust from a moving image for the full recording time, the dust position is unchanged between frames. However, an image around the dust shadow-containing region changes. That is, an image around the dust shadow-containing region differs between frames. A resultant image becomes unnatural not only in each frame like a still image, but also between successive frames.
For example, FIG. 15 is a view showing six frames 1 to 6 which are extracted from a moving image recorded for an arbitrary time and arranged in time series. In FIG. 15, the dot mark (•) represents dust. This moving image is captured while panning the camera rightward with respect to two ropes indicated by solid lines.
When dust removal processing is performed for a moving image as shown in FIG. 15, dust shadows in frames 1, 4, and 6 are interpolated (the ∘ mark of a broken line represents a corrected dust shadow), and those in the remaining frames are not interpolated. When the moving image in FIG. 15 is played back, the shadow of dust at the same position between frames is corrected and disappears in some frames, but is not corrected and appears in other frames, as shown in FIG. 16. The dust removal processing makes a moving image unnatural.